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Abstract 

We present a general framework for discriminative estimation based on the maximum en¬ 
tropy principle and its extensions. All calculations involve distributions over structures and/or 
parameters rather than specific settings and reduce to relative entropy projections. This holds 
even when the data is not separable within the chosen parametric class, in the context of 
anomaly detection rather than classification, or when the labels in the training set are uncer¬ 
tain or incomplete. Support vector machines are naturally subsumed under this class and we 
provide several extensions. We are also able to estimate exactly and efficiently discriminative 
distributions over tree structures of class-conditional models within this framework. Preliminary 
experimental results are indicative of the potential in these techniques. 


1 Introduction 

Effective discrimination is essential in many application areas including speech recognition, im¬ 
age classification or identification of molecular binding sites in genomic DNA. Statistical approaches 
used in these contexts for classification generally fall into two major categories - generative or dis¬ 
criminative - depending on the estimation criterion used for adjusting the model parameters and/or 
structure. Generative approaches rely on a full joint probability distribution over examples and clas¬ 
sification labels whereas for discriminative methods only the conditional relation of a label given the 
example is relevant. While the full joint distribution in the generative approach carries a number of 
advantages e.g. in handling incomplete examples, the typical estimation criterion (maximum likeli¬ 
hood or its variatiants) is nevertheless suboptimal from the point of view of classification objective. 
Discriminative methods such as support vector machines [21] or boosting algorithms [8] that focus 
directly on the parametric decision boundary typically yield more robust classification methods, 
whenever they are applicable. 

Full joint distributions and the benefits they convey can be, of course, exploited in discriminative 
approaches as well. We may, for example, interprete the posterior probability of a label given the 
example as a parametric decision boundary (see e.g. [10, 13]). Alternatively, we can induce suitable 
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vector space representations for examples from generative models and feed such representations into 
standard discriminative techniques [ 11 ]. 

In this paper we provide a more general notion of discrimination, one that applies also in the 
contex of anomaly detection or when the classification labels themselves are uncertain or missing. 
Note that the utility of e.g. unlabeled examples is not obvious [22, 2, 4, 18]. Our approach towards 
general discriminative training relies on the well known maximum entropy principle which embodies 
the Bayesian integration of prior information with observed constraints (see e.g. [15]). The formalism 
that we apply and extend in this paper allows, for example, a feasible discriminative training of both 
the parameters and the structure of a class of joint probability models. The approach is not limited 
to probability models, however, and we extend e.g. support vector machines. 

2 Maximum entropy classification 

Consider first a two-class classification problem where labels y G { — 1,1} are assigned to examples 
A' £ X. Assume we have two class-conditional probability distributions over the examples, i.e., 
P(X\6y) with parameters 6 V , one for each class. The decision rule corresponding to any particular 
parameter setting {#± 1 } follows the sign of the discriminant function: 

ax ' e) = k ’*WdB) +b (1) 

where 0 = {6i,9-i,b} and b is a bias term, usually expressed as a log-ratio of prior class probabilities 
b = logp/(l — p) . The class-conditional distributions here may come from different families of 
distributions or we might specify the parametric discriminant function directly without any reference 
to probability models. The parameters 6 y may also include the model structure as seen later in the 
paper. 

The parameters 0 = {6i,6-i,b} in the discriminant function should be chosen to maximize 
classification accuracy. Instead of finding a single parameter setting, we consider here a more 
general problem of finding a distribution P(0) over the parameters and using a convex combination 
of discriminant functions, i.e., 


J P{Q)jC(X\Q)dQ ( 2 ) 

in place of the original discriminant function in the decision rule. The problem is now to find an 
appropriate distribution P(0). Given a set of training examples {X%,... ,Xt} and corresponding 
labels {yi ,..., yx} we seek for a distribution P(0) that makes the least assumptions about the choice 
of the parameter values 0 while giving rise to a discriminant function that correctly separates the 
training examples. We can formalize this as a maximum entropy (ME) estimation problem. In other 
words, we maximize the entropy H(P) of P subject to the classification constraints 

J P(Q)[y t X(X t \Q)]dQ>j (3) 

for all t = 1,... ,T. Here 7 specifies a desired classification margin. We note that the solution is 
unique (provided that it exists) since P(P) is concave and the linear constraints specify a convex 
region. Note that the preference towards high entropy distributions (fewer assumptions) applies 
only within the admissible set of distributions P 7 consistent with the classification constraints. 

We can readily extend this formulation to a multi-class setting by introducing additional clas¬ 
sification constraints. To see this, suppose we have instead m class-conditional probability models 
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P(X\8 y ), y = 1, .-.. , m, prior class frequencies {py}, and the associated pairwise discriminant func¬ 
tions 


£ y ,y'{X t \Q) = log 


P(X\6y) 

p(x\ey) 


+ log — 

Py' 


(4) 


where 0 = { 6 1 ,... ,0 m ,p\,... ,p m }. We may now replace the single constraint per training example 
in eq. (3) with the following m — 1 pairwise constraints 


I P(0) [ Cy t ,y(X t \Q) ] d& >7, y? y t , (5) 

to ensure that the training label y t always “wins” the competition against the alternative labels 
y ^ yt- For notational simplicity we will consider primarily only binary classification problems in 
the remainder of the paper but emphasize that the analogous extension to a multi-class setting can 
be made. 

The overall ME formulation presented so far has several problems. We have, for example, made 
a tacit assumption that the training examples can be separated with the specified margin. This 
assumption may very well be violated in practice. Moreover, we may have a prior reason to prefer 
some parameter values over others (as well as margin constraints) which requires us to incorporate a 
prior distribution P o (0, 7 ) into the definition. Other extensions and generalizations will be discussed 
later in the paper. 

A more general formulation that addresses these concerns is given by the following minimum 
relative entropy principle: 

Definition 1 Let {Xt, yt} be the training examples and labels, £(A'|0) a parametric discriminant 
function, and 7 = [ 71 , • • ■, 7 t] a set of margin variables. Assuming a prior distribution Po(0,7), we 
find the discriminative minimum relative entropy (MRE) distribution P( 0 , 7 ) by minimizing 

P(P||Po) = /p(0)log^|d0 ( 6 ) 

subject to the (soft) classification constraints 

j P(Q, 7 ) [Vt £(A'i| 0 ) - 71 ] dQd 1 > 0 (7) 

for all t. The decision rule for any new example X is given by 

V = sign ^ J P( 0 ) P(W| 0 ) d@ ^ ( 8 ) 


Let us make a few remarks about the definition. First, we can recover the previous ME formula¬ 
tion by appropriately adjusting the prior distribution Po(0, 7 ) (e.g., if Po( 7 ) peaks around a specific 
setting of the margins). It is clear that the margin constraints are hidden in the prior distribution 
Po( 7 ). Second, if we assume that there is a non-zero prior probability for all 7 1 taking some negative 
values, we guarantee that the admissible set V composed of all distributions P(0, 7 ) consistent with 
the classification constraints, is never empty. Thus even when the examples cannot be separated 
by any discriminant function in the chosen parametric class (e.g. linear), we get a valid and unique 
solution. Third, the penalty for violating any of the margin constraints also depends on the prior 
distribution Pq ; whenever the mean of 7 1 deviates from its prior mean under Pq , we incur a penalty 
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Figure 1: Minimum relative entropy (MRE) projection from the prior distribution to the admissible 
set. 

in the form of relative entropy distance between the corresponding distributions. It is worth noting 
that the penalties are defined in terms of joint specifications of margins but, in certain cases, they 
reduce to the more typical additive penalties of violating the constraints. 

The prior Po(0, 7 ) plaves an important role in our definition and we must choose it appropriately. 
Let us consider here only the prior over the margin constraints 7 . Supposing again that Po(0,7) = 
Po(Q)Po(j), we can, for example, set 


Po(7)=n^) ( 0 ) 

t 

where Poi'jt) = for < 1 . A penalty is incurred for margins smaller than 1 — 1 /c (the 

prior mean of 7 1 ) while margins larger than this are not penalized. In the latter case, the associated 
constraint becomes merely irrelevant. We will see in later sections that this choice of the margin 
prior corresponds closely to the use of slack variables and additive penalties used in support vector 
machines. A number of other choices for P 0 ( 7 ) are possible and we discuss some of them later in 
the paper. 

An important property of the MRE solution is that it can be viewed as a relative entropy 
projection, the e-projection in the terminology of [1], from the prior distribution P 0 (Q,j) to the 
admissible set V. Figure 1 illustrates this view. Even in the non-separable case, we can view the 
MRE solution as a projection. This formalism readily extends to the case of uncertain or partially 
labeled examples as we will see later in the paper. 

To solve the MRE problem, we rely on the following theorem. 

Theorem 1 The solution to the MRE pivblem has the following general form (cf. [7]): 

/m7) = ^/;,((-): 7)' "pb v < e ; •*' (io) 

where Z( A) is the normalization constant (partition function) and A = {Ai,...,Ay} defines a set 
of non-negative Lagrange multipliers, one for each classification constraint. A are set by finding the 
unique maximum of the following jointly concave objective function: 

J(\) = -\ogZ(\) (11) 


Whether the MRE solution can be found in a feasible way depends entirely on whether we can 
evaluate the partition function Z( A), 


Z( A) = j Po(0,7)e^ At[y,£( - Y(|e) ~ 7,] d0d 7 


( 12 ) 
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in closed form. Given a closed form expression for Z( A), the maximum of the jointly concave ob¬ 
jective function .J( A) can be subsequently found through any standard convex optimization method 
such as Newton-Raphson. The resulting set of Lagrange multipliers {A t } then define the MRE 
solution as indicated in the theorem. Finally, predicting a label for any new example X involves av¬ 
eraging the discriminant function £(0) with respect to the marginal P(0) of the MRE distribution 
(see Definition 1). Finding this marginal as well as performing the required averaging are no more 
costly than computing Z( A). We will elaborate these calculations further in the context of specific 
realizations. 

The MRE solution is sparse in the sense that only a few Lagrange multipliers will be non-zero. 
This arises because many of the classification constraints become irrelevant once the constraints are 
enforced for a small subset of examples. For support vector machines that are subsumed under the 
above general definition, this notion translates into a sparse representation of the separating hyper¬ 
plane. Sparsity leads to immediate generalization guarantees (independent of the dimensionality of 
the parameter or example space): 

Lemma 1 The generalization error e g of the MRE classifier satisfies 

e g < E{ fraction of non-zero Lagrange multipliers } (13) 

where the expectation is over the choice of the training set. 

Practical leave-one-out cross-validation estimates of the generalization error can be derived on 
the basis of this result (cf. [21, 12]). We may also make use of generalization error results derived 
for convex combination of classifiers [20] to obtain more informative generalization bounds for MRE 
classifiers. The details are left for another paper. 

3 Practical realization of the MRE solution 

We now turn to the question of actually finding the MRE solution. Consider first the following 
elementary but helpful lemma 

Lemma 2 Any factorization of the prior Po(0,7) across any disjoint sets of variables { 6 , 7 } leads 
to a disjoint factorization of the MRE solution P(0,y) across the same sets of variables provided 
that these variables appear in distinct additive components in ytC(Xt,Q ) —7 1 - 

If we assume that the labels {y t } are fixed and that the prior distribution P<j(0,7) factorizes 
across the components {0\6, b, 7 }, then according to the lemma, the MRE solution factorizes in the 
same way. This factorization property allows us to eliminate e.g. the bias term from the remaining 
solution by means of imposing additional constraints on the Lagrange multipliers. This is analogous 
to the handling of the bias term in support vector machines [ 21 ]: 

Lemma 3 Assuming Po(0,7) = Po(0 \ b,j)Po(b) and Po(b) approaches a non-informative prior, 
then P(0, 7 ) = P(0 \ b, r y)P(b ) and P(0 \ b , 7 ) can be found independently from P(b ) provided that 
we require \tyt = 0. 

With the help of these results, we will consider now a few specific realizations such as support 
vector machines and a class of graphical models. 
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Figure 2: Three margin prior distributions (top row) and the corresponding potential terms (bottom 
row) from Eq. (15). 


3.1 Support vector machines 

It is well known that the log-likelihood ratio of two Gaussian distributions with equal covariance 
matrices yields a linear decision rule. With a few additional assumptions, the MRE formulation 
gives support vector machines: 

Theorem 2 Assuming C{X,Q) = 6 T X — b and Pq{Q,j) = Po{9)Po{b)Po{'j) where Po{9) isN(0,I), 
Po(b) approaches a non-informative prior, and Po(j) is given by eq. (9) then the Lagrange multipliers 
X are obtained by maximizing J( A) subject to 0 < At < c and Xpyt = 0, where 

j( A) = XXA t + log(l - A t/c)} - ^tAt'Vtyt' (X?X t> ) (14) 

t t,t' 


The only difference between our J( A) and the (dual) optimization problem for SVMs is the 
additional potential term log(l — A t /c). This highlights the effect of the different miss-classification 
penalties, which in our case come from the MRE projection. Figures 2a) and c) show, however, 
that the additional potential term does not always carry a huge effect (for c = 5). Moreover, in the 
separable case, letting c —>■ oo, the two methods coincide. The decision rules are formally identical. 

The choice of the prior distribution Po(-y) leads to different potential terms. Figure 2 gives the 
following priors and their corresponding potential terms 

Margin prior Dual potential term 

a) Foil) oc e -c ( 1-7 ),7 < 1, X t + log(l - X t /c) 

b) Poh) oce^l 1 -^ 1 , Xt + 2 log(l — Xt/c) 

C ) Po( 7 ) e~ c2 f 1 -^) 2 / 2 , At - {Xt/c) 2 

where a) is the case discussed in the theorem. Note that the resulting potential terms may or may 
not set an upper bound on the value of A t- In a) and b) At is bounded by the constant c whereas 
in c) no such bound exists. 
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3.1.1 Extension 

We now consider the case where the discriminant function C(X, 0) corresponds to the log- 
likelihood ratio of two Gaussians with different (and adjustable) covariance matrices. The parame¬ 
ters 0 in this case are both the means and the covariances. The prior P o (0) must be the conjugate 
Normal-Wishart to obtain closed form integrals 1 for the partition function, Z. Here, P(0i,0_i) 
is P(mi,Vi)P(m-i,V-i), a density over means and covariances (and the factorization follows from 
our assumptions below). 

The prior distribution has the form P o (0i) = jV(mi;mo, Vi/k) XW(Vj; kVo,k) with parameters 
(fc, mo, Vo) that can be specified manually or one may let k 0 to get a non-informative prior. We 
used the MAP values for k , mo and Vo from the class-specific data 2 . Integrating over the parameters 
and the margin, we get a partition function which factorizes Z = Z 7 x Zi x Z_i. For Z\ we obtain 
the following: 


Z 1 oc Aj ri/ ' 2 |7rSj| _iVl/2 nj = |F ^ + 2 1 —(16) 

Ni = J2t w t A'i=E Si = Ei w t X t Xf - NiXiXf (17) 

Here, w t is a scalar weight given by w t = u{yt ) + ytM for Z\. To solve for Z_i we proceed in a 
similar manner with the exception that the weights are set to w t = u{—y t ) — yt Af. u(-) here is the 
step function. Given Z, updating A is done by maximizing the corresponding negative log-partition 
function J( A) subject to 0 < A f < c and Ylt X tVt = 0 where: 

J( A) = +l°g(l - E/c)] — log Zt (A*) — log Z-! (At) (18) 

t 

The potential term above corresponds to integrating over the margin with a margin prior P 0 ( 7 ) oc 
e -c(i a -i) w j^i 1 j <l a . We pick l a to be some a-percentile of the margins obtained under the standard 
MAP solution. Optimal lambda values are found via constrained gradient descent. The resulting 
marginal MRE distribution over the parameters (normalized by the partition function Zi x Z_i) is 
a Normal-Wishart distribution itself, P(0i) = A’(//h !• A 1 -1 ’1 /Aj I PW(Vj; Si , Ni) with the final A 
values. Predicting the labels for a data point X under the final P(0) involves taking expectations 
of the discriminant function under a Normal-Wishart. This is simply: 

E Piei) [logP(X\Qi)] = constant - ^-(X - Xi) T S^(X - Xi) (19) 

We thus obtain discriminative quadratic decision boundaries. These extend the linear boundaries 
without (explicitly) resorting to kernels. Of course, kernels may still be used in this formalism, 
effectively mapping the feature space into a higher dimensional representation. However, unlike 
linear discrimination, the covariance estimation in this framework allows the model to adaptively 
modify the kernel. 

3.1.2 Experiments 

In the following, we show results using the minimum relative entropy approach where the dis¬ 
criminant function (£(X, 0)) is the log-ratio of Gaussians with variable covariance matrices on 
standard 2-class classification problems (Leptograpsus Crabs and Breast Cancer Wisconsin). In 

1 This can be done more generally for conjugate priors in the exponential family. 

2 The prior here is the posterior distribution over the parameters given the data, i.e. an empirical Bayes procedure. 
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Method 

Training 

Errors 

Testing 

Errors 

Neural Network (1) 


3 

Neural Network (2) 


3 

Linear Discriminant 


8 

Logistic Regression 


4 

MARS (degree = 1) 


4 

PP (4 ridge functions) 


6 

Gaussian Process (HMC) 


3 

Gaussian Process (MAP) 


3 

SVM - Linear 

5 

3 

SVM - RBF o- = 0.3 

1 

18 

SVM - 3rd Order Polynomial 

3 

6 

Maximum Likelihood Gaussians 

4 

7 

MaxEnt Discrimination Gaussians 

2 

3 


Table 1: Leptograpsus Crabs 





(a) Training ROC (b) Testing ROC 


Figure 3: ROC curves on Leptograpsus Crabs for discriminative (solid line), Bayes / ML models 
(dashed line) and SVM linear models (dotted line). 


addition we display a two-dimensional visualization example of the classification. Performance is 
compared to regular support vector machines, maximum likelihood estimation and other methods. 

The Leptograpsus crabs data set was originally provided by Ripley [19] and further tested by 
Barber and Williams [3]. The objective is to classify the sex of the crabs from 5 scalar anatomical 
observations. The training set contains 80 examples (40 of each sex) and the test set includes 120 
examples. 

The Gaussian based decision boundaries are compared in Table 1 against other models from[3]. 
The table shows that the maximum entropy (or minimum relative entropy) criterion improves the 
Gaussian discrimination performance to levels similar to the best alternative models. The bias was 
estimated separately from training data for both the maximum likelihood Gaussian models and the 
maximum entropy discrimination case. In addition, we show the performance of a support vector 
machine (SVM) with linear, radial basis and polynomial decision boundaries (using the Matlab 
SVM Toolbox provided by Steve Gunn). In this case, the linear SVM is limited in flexibility while 
kernels exhibit some over-fitting. 

In Figure 3 we plot the ROC curves on training and testing data. The ROC curve shows improved 
classification for maximum entropy (minimum relative entropy) case. 





























Method 

Training 

Errors 

Testing 

Errors 

Nearest Neighbour 


ii 

SVM - Linear 

8 

10 

SVM - RBF a = 0.3 

0 

11 

SVM - 3rd Order Polynomial 

1 

13 

Maximum Likelihood Gaussians 

10 

16 

MaxEnt Discrimination Gaussians 

3 

8 


Table 2: Breast Cancer Classification 



(a) Training ROC (b) Testing ROC 


Figure 4: ROC curves on Breast Cancer for discriminative (solid line), Bayes / ML models (dashed 
line) and SVM linear models (dotted line). 


Another data set which was tested was the Breast Cancer Wisconsin data where the two classes 
(malignant or benign) have to be computed from 9 numerical attributes from the patients (200 
training cases and 169 test cases). The data was first presented by Wolberg [24]. We compare our 
results to those produced by Zhang [25] who used a nearest neighbour algorithm to achieve 93.7% 
accuracy. As can be seen from Table 2, over-fitting seems to prevent good performance for kernel 
based SVMs. The maximum entropy discriminator achieves 95.3% accuracy. 

In Figure 4 we plot the ROC curves on training and testing data. The training ROC curves 
show improved discrimination for the maximum entropy method. ROC curves for all three methods 
are equivalent on testing however since we typically assume that bias is estimated exclusively from 
training data, the results in Table 2 are more significant. 

Finally, for visualization, we present the technique on a 2D set of training data in Figure 5 and 
Figure 6. The SVM in Figure 5(a) attempts to achieve maximum descrimination but is limited to a 
linear decision boundary. It only succeeds after the application of a kernel as in Figure 5(b), where 
a 3rd order polynomial kernel is used. In Figure 6(a), the maximum likelihood technique is used 
to estimate a 2 Gaussian discrimination boundary (bias is estimated separately) which has more 
flexibility than the linear SVM yet fails to achieve the desired optimal classification. Meanwhile, 
the maximum entropy discrimination technique places the Gaussians in the most discriminative 
configuration as shown in Figure 6(b). 
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(a) Linear SVM (b) Polynomial Kernel SVM 

Figure 5: Classification visualization SVMs. 



(a) Max Likelihood (b) Max Ent Discrimination 

Figure 6: Classification visualization for Gaussian discrimination. 


3.2 The Fisher kernel classifier 

Here we demonstrate that the MRE formulation proposed in this paper contains the Fisher kernel 
method of [11]. The Fisher kernel method provides a combination of a generative model P(X\8) 
with a discriminative method such as support vector machines through defining an appropriate 
kernel function. The kernel function, called the Fisher kernel, can be computed from any generative 
model in the neighborhood of some desired e.g. maximum likelihood parameter setting 9*. The 
Fisher kernel function is given by 

K fk (X,X') = U X (9*) T FiO*)- 1 U X '(9*) (20) 

where Ux{9) is the Fisher score 


U x (8) = V fl logP(A'|0)| e=fl ,, 


( 21 ) 
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F(8) = E{ Ux(0)l'x(0) } is the Fisher information matrix 3 and the expectation is with respect to 
P(X\6). Replacing the inner product XfXf between the examples in Theorem 2 with the kernel 
function in Eq. (20) amounts to the “simple” Fisher kernel method as explained in [11]. 

Our goal in this section is to show that we can recover the Fisher kernel method in the MRE 
framework so long as the prior distribution Po(6, 7 ) is chosen in an appropriate way. We start with a 
few necessary regularity assumptions about the family of distributions P(X\0) in some small (open) 
neighborhood 0(8*) of 8*: 

1. for any A' € X, Ux(8) = Xg logP(A'|$) is a continuously differentiable vector valued function 
of 8 


2. F{9) = E{ Ux(8)U\(6) } exists and is positive definite 


Let us define, in addition, the differential (symmetric) relative entropy distance between the 
distributions P(X\8) and P(X\8*) 

d(8,8*f = i(<9 - 8*) t Fid*)- 1 (8-8*) ( 22 ) 

valid whenever 8 m 8* . We assign a prior distribution Pq(8) in terms of this distance 4 


Po(0) 


1 -pd(e,e-) 2 

Z(8*, 8 ) 


(23) 


where J3 serves as a scaling parameter. This prior assigns a low probability to all 8 for which the 
corresponding probability distribution P(X\6) deviates significantly from P(A"|#*). Another way to 
view this prior is as a local isotropic Gaussian prior distribution in the probability manifold induced 
by the family of distributions P(X\8), 8 £ 0(8*). 

In the MRE formalism the objective is to minimize the relative entropy distance between the 
MRE distribution P and the prior P 0 subject to the classification constraints 

I P(0, 7 ) [yt £(At|0, 0) — It] dPtdry > 0 (24) 


where the discriminant function £(A' t |0, /3) is the scaled log-likelihood ratio: 


C(X t \Q,!3) 


[/3 1/2 


log 


P(X t \8*) 


b] 


(25) 


and 0 = {8, b}. This discriminant function encourages parameter values 8 that are indicative of the 
+ 1 class relative to the “null model” P(X t \6*). 

The following Theorem now establishes the desired connection to the Fisher kernel method. 


Theorem 3 If we replace Pq( 8) with Eq. (23) in Theorem 2 and the discriminant function with 
C(Xt |©, /?) defined above as well as let fi —f 00 , then the objective function J( A) reduces to 

J( A) = £[A t +log(l - A t /c)]~ lY, *tXt>VtVt' K fk( x u AV) (26) 

t Z l.l' 

where Kfk(Xt,X?) is the Fisher kernel of Eq. (20). 

We note that this result is merely a formal relation between the MRE principle and the Fisher 
kernel and does not necessarily provide any additional motivation. 

3 For many probability distributions the Fisher information matrix may not be possible to compute in closed form. 
However, it is the covariance matrix of the Fisher scores and thus can be easily approximated by sampling. 

4 A more precise definition of this prior would involve setting it to zero outside the open neighborhood where the 
regularity conditions may no longer hold. For large /3, the effect of this condition vanishes and we omit it here for 
simplicity. 
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3.3 Graphical models 


The MRE formulation can accomodate discriminant functions resulting from log-ratios of general 
graphical models. The MRE distribution, i.e. P(0), in this setting is over both the parameters 
and the structure of the model. Since the estimation is carried out in the space of distributions 
the distinction between discrete or continuous variables is immaterial. The framework does not, 
however, admit efficient solutions without restrictions on the class of graphical models. For example, 
assuming the structure remains fixed and that the class-conditional models have no latent variables, 
then the MRE distribution P(0) over the parameters can be obtained efficiently. This requires 
additional technical assumptions such as the use of conjugate priors, the parameter independence 
assumption of [6] and the fact that the probability model must be tractable for any fixed setting 
of the parameters. Although restricted, this class does include e.g. naive Bayes models, mixture of 
tree models and so on. 

For a special class of graphical models whose structure is a tree, both the parameters and the 
structure can be estimated efficiently within our discriminative framework. In the remainder, we 
will consider such tree structured models. 

First, we define a tree distribution. Let V denote the set of variables of interest, | Vj = n, x v G X v 
a particular value of v G V and X G X an assignment to all the variables in V. Like any graphical 
model, a tree distribution is defined in two stages. First, one defines a graph (V, E), called structure , 
whose vertices are the variables in V and whose edges encode dependencies between these variables. 
A tree is an undirected graph over V that is connected and has no cycles. For any tree over n 
vertices \E\ = n — 1. Because such a tree spans all the nodes in V, it is often called a spanning 
tree. Then, the tree distribution is defined as a product of factors corresponding to the edges and 
vertices. 


_ n^wjg® t uv { x u , x v ) 


(27) 


where degu is the degree of vertex v, i.e. the number of edges incident to v G V and T uv and T v 
denote the marginals of T: 


i'u v (Xu . X v ) 


T v (x v ) 


E 

V = X v ,U=X u 

E T W- 

V = X v 


When the variable x is discrete, the marginals T uv and T v can be represented as probability tables 
denoted respectively 6 uv (x u , x v ) and 0 v {x v ). The values 0 are the parameters of the distribution. 
When it will be necessary to emphasize the dependence of the tree distribution on its structure and 
parameters we will use the notation T(x\E,6). 

By taking the logarithm of T( X) and conveniently grouping the factors one obtains 


log TO 


E lo S T v (x v )+ E lo 6 

vGV uvGE v 

v -v-' 

wo(X) 


Tuv {%u 5 %v) 

T v (x v )T u (x u )' 




W 0 (X)+ E w uv(X)- 

uv£E 


(28) 


In words, the log-likelihood is a sum of terms w uv (X) each corresponding to an edge (and depending 
only on the values of the variables u,v associated with that edge) plus a structure independent term 
wq(X) that depends on all the variables. All the terms are functions of the tree parameters 8. 
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3.3.1 Discriminative learning of tree structures 

A tree model is defined by a set of discrete variables encoding its structure and a set of continuous 
variables representing its parameters. To use the MRE framework we must define a prior joint distri¬ 
bution over the structures and their associated parameters. We will assume that the structure and 
the parameters are independent a priori; moreover, we shall assume that except for the functional 
dependencies among the parameters that are imposed by the fact that they have to represent a valid 
joint distribution over X there are no other statistical or functional dependencies. These assumptions 
correspond to the parameter independence and parameter modularity assumptions of [9] (see also [ 6 ]). 
In our case, this means that there is a set of parameters 9 = {6 uv (i, j),u,v G V, i G X u ,j G X v } 
associated with the edges such that in any tree model containing an edge uv G E , the pairwise 
marginals T uv {x u ,x v ) are given by 9 uv (x u ,x v ) regardless of the presence of other edges in E and 
their parameter values. This simplification, in turn, allows the MRE formulation for only structures 
(with a fixed set of parameters or a fixed distribution over their values), for parameters only, or for 
both. 

We start with a MRE estimation of structures only when the pairwise marginals 9 uv (x u , x v ) are 
assumed fixed. Note that each tree nevertheless makes use of a different set of n — 1 edges and 
thereby a different set of parameters. For each class or label s G {1,-1}, we have a separate set of 
fixed parameters 6 s . In the experiments below, the values of these parameters were obtained from 
empirical (class-conditional) marginals. We assume a uniform prior over the class-conditional tree 
structures E s . 


Definition 2 Given a set (X t ,y t ),t = 1 ,...T of labeled examples, a set of margin vaiiables 7 = 
[ 71 ,... ,7 t] and a prior distribution Po(Ei , F_i, 7 ) the MRE distribution P(E\ , P_i, 7 ) is the one 
minimizing D(P\\Pq) subject to 


Y [ /’(/••'i./-.' i-7) 


TjXtlE^e 1 ) 

TiXtlE-uO- 1 ) 



c /7 > 0 for t = 1,... T 


(29) 


Assuming Po{Ei, E-m) = Po(Ei)P 0 (E_i)P 0 (”/), Lemma 2 implies that the solution is factored as 
P(Ei)P(E_i)P( 7 ) with 


P(Es) 



sX tVt[uC 0 (Xt) + ^ iuh€E ^ 



n " 

uv£E s 


(30) 


for s = 1,-1 and 


Wo 


P^lt sX lUt w o( X i) TT/ S 
° ’ * v u\ 


T 


t =1 


5 = 1,-1. 


(31) 


In the above the normalization constants Z s and the factors W s are functions of the Lagrange 
multipliers A which need to be set. Provided that we can obtain the normalization constants 
(functions) Z s in closed form, A are set to maximize the dual objective 


J( A) = 7 • A - logZi - logZ_i. 


(32) 


where, for simplicity, we have assumed a fixed setting of the margin variables { 7 4 }. 


3.4 Computing the normalization constant and its derivatives 

The number of all possible tree structures over n vertices is n "~ 2 [23] and thus computing the 
normalization constants by enumerating all the tree structures is clearly not possible for reasonable 
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n. However, a remarkable graph theory result enables us to perform all the necessary summations 
in closed form in polynomial time. This is the Matrix Tree Theorem quoted below. 


Theorem 4 (Matrix Tree Theorem)[23] Let G = (V, E) be a multigraph and denote by a uv = 
a vu > 0 the number of undirected edges between vertices u and v. Then the number of all spanning 
trees of G is given by |A|„„( — l) («+«) £/j e va l ue of the determinant obtained from the following matrix 
by removing row u and column v°. 


deg(vl) -ai2 —a 13 • • • -«i,n 

-021 deg(rio) -«23 • • • -o 2 ,„ 


A = 


(33) 


^n, 1 


2 


deg(v n ) 


By extending the Matrix Tree theorem to continuous-valued A and letting the weights W uv play 
the role of a uv , one can prove 

Theorem 5 Let P(E ) be a distribution over tree structures defined by 

P(E ) oc Wo n w ™ (34) 

uv£E 

Then its normalization constant Z is equal to 

Z = Wo En w uv = Wo\Q(W)\ (35) 

E uv£E 


with Q(W) being the ( n — 1) x (n — 1) matrix 


Quv(W) 



1 < v < v < n — 1 
1 < v = v < n — 1 


(36) 


This shows that summing over the distribution of all trees, when this distribution factors according 
to the trees’ edges, can be done in closed form by computing the value of a order n — 1 determinant, 
operation that involves 0(n 3 ) operations. 

To optimize the Lagrange multipliers, we must compute derivatives of J(A) or, equivalently, 
derivates of the log-partition functions with respect to A. It is well known that such derivatives lead 
to averages with respect to the distribution in question (for details see Appendix A). In our case, 
for example, 


0 log 
dX t 


syt < log T(X t \E s , 9 s ) >p(e b ) 


sy 


<,( V') + E K 

U^V 


(37) 


where M s is a linear function of Q(W s ) given in Appendix A. Inverting the matrix Q(W) is 0(n 3 ) 
and this operation can be done once before the summations in equations (37). Thus, computing 
the derivatives of the normalization constant w.r.t all X t takes 0(n 3 + n' 2 T) operations and 0(n 2 ) 
extra space. 

J Note that A as a whole is a singular matrix. 
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Finally, to obtain the decision rule for any new example X we must compute averages of the 
log-likelihood ratio with respect to the (marginal) MRE distribution | ): 

V = sgn{ E Ei ,e^ W)^-i)logTTS^&) } (38) 

= sgn{ wl(X)-Wo 1 (X) + <^2wlv{X)>p^i - <E Kv( x )>p(E-i) } 

uv£E i uv£E-i 

(39) 

where we have omitted a possible bias term b. The required averages can be computed analogously 
to Eq. (37) yielding e.g. 

<E wl (.Y ( X)W m Ml (40) 

uv£Ei u^v 

where M^ v is the same matrix as in Eq. (37) and has been already computed in the last step of the 
training algorithm. Classifying a new data point therefore requires only roughly 0{n 2 ) operations. 

3.5 MRE distributions over tree structures and parameters 

Here we describe briefly how to find the MRE distribution over both structures and parameters, 
i.e., P(Ei,0 1 ,E_i,0 _1 ). We assume a factored prior Po(d 1 )Po(8~ 1 ) over the parameters and as be¬ 
fore a uniform prior over the structures. In addition to the parameter independence and modularity 
assumptions used earlier, we must assume that the priors Po(0 s ), s = 1, — 1 are likelihood equivalent 
(i.e. they assign the same value to models having the same likelihood for all data sets). In this case, 
the priors over parameters are forced to be Dirichlet [9] and defined in terms of a set of equivalent 
marginal counts N^ v (x u ,x v ) satisfying 

E = K(x v ) ■) = K(X U ) E K V (X U ,X V ) = N s (41) 

x u x v x u x v 

Because the prior over parameters is independent of the structure, the MRE distribution factor¬ 
izes as 

P(E S ,6 S ) = l-P 0 (e 8 )e^t sXtVtlogT{XAE - r) (42) 

Zs 

To evaluate the partition function Z s , the parameters 8 s can be analytically integrated out before 
the summation over structures. The resulting marginal distribution over tree structures is similar 
to equation (35) 

P(Es) = f Eli W ™ (43) 

s E uvGE 

with the factors W s are now functions of both A and Dirichlet distribution parameters N s (see 
appendix B for exact expression). 

The classification rule is also similar in form to equation (39) with the terms w s depending on 
A, the data, and the equivalent counts as described in Appendix B. 

3.6 General Bayes nets 

A Bayes net with given structure can be parametrized by the set of conditional distributions 
P(v |pa(n) = x p& ( V )) of a variable given a configuration of its parents. A discriminative MRE solution 
can be found for the parameter distribution P)# 1 ,# -1 ) assuming complete observations. Finding 
the MRE distribution over structures is, however, unlikely to be feasible for other than trees (c.f. 

[5])- 
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Figure 7: ROC curves for the ME discriminative classifier (full line) and the ML classifier (dashed 
line) for the splice junction classification problem. The minimum test errors are 12.4% and 14% 
respectively. 
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Figure 8 : Logarithmic weights w uv versus mutual informations I uv for class 1 (a) respective —1 (b). 
The square in position uv, u < v represents w uv while its symmetric, vu represents I uv . Larger 
values appear more back in the figures. 


3.7 Experiments 

We tested our model in the fixed parameter version on the detection of DNA splice sites and 
compared its performance to the performance of a classifier using a Maximum Likelihood (ML) tree 
for each class. In both cases, the tree parameters 6 were the ML parameters for the corresponding 
class (empirical class-conditional marginals). 

The domain consists of 25 variables representing sites around a (hypothetic) splice junction. 
The test set had 400 examples split equally between the two classes; the training set consisted of 
4724 examples, about a fourth being positives ones. For simplicity, we used a fixed margin 7 = 4, 
the largest value that allowed perfect class separation. The number of A’s that are nonzero in this 
example is 61 (out of 400) suggesting a performance level of about %15 according to Lemma 1. The 
ROC curves for the two classifiers are compared in figure 7. MRE distribution over tree structures 
is superior to a pair maximum Likelihood trees, although the parameter values are identical. The 
test set error is 14.0% for the ML classifier and 12.3% for the MRE method. The training error is 
0.5% for the ML classifier and zero for the discriminative one indicating that the MRE method is 
resistant to overfitting. 

Figure 8 compares the “edge weights” for the two classifiers. These edge weights reflect the 
preferences assigned to tree structures in the MRE distribution or in the (single) class-conditional 
maximum likelihood (ML) tree. Since the estimation criterion differs in the two cases, the most 
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likely tree in the MRE solution does not in general equal the ML tree structure. Figure 8 a) displays 
w\ v = log(W^) factors corresponding to each edge uv in the MRE distribution for class 1 as well as 
the respective mutual information values I* v . Since both matrices are symmetric, one can display 
both sets of values in a 25 by 25 square: the upper left half represents the ME weights whereas the 
lower right half of the square shows the mutual information. Figure 8 ,b shows the same results for 
class -1. Note that summing w\ v or I^ v across the edges of a particular tree pertains directly to the 
log-probability of the tree and thus the comparison is meaningful 6 . 

The figure shows that there are relatively few edges with large weights on both sides of the 
diagonal. This is particularly relevant for the discriminative model of the positive examples, since 
it shows that the MRE distribution decays rapidly around its peak. The maximum W^ v is more 
than 10 3 times the next largest value, clearly separating edges that are discriminative and those 
whose inclusion or exclusion has little effect on discrimination. This contrast is understandably less 
pronounced for the negative examples that represent a diverse collection of spurious splice sites. 

A second important remark is that neither figure 8 ,a nor 8 ,b are symmetric w.r.t the diagonal. In 
other words, not all pairs of variables that exhibit high mutual information are also discriminative. 
Note for example that the subdiagonal band showing that adjacent variables are informative of 
each other is almost completely effaced under discriminative training. Our method brings out the 
discriminative structure of the data, which is different from its structure as a density estimator. 

4 Anomaly detection 

In anomaly detection we are given a set of training examples representing only one class, the 
“typical” examples. We attempt to capture regularities among the examples to be able to recognize 
unlikely members of this class. Estimating a probability distribution P(X\0) on the basis of the 
training set {Ah,,... Ah} via the standard maximum likelihood (or analogous) criterion is not 
appropriate since there is no reason to further increase the probability of those examples that are 
already well captured by the model. A more relevant measure involves the level sets 

X 1 = {X EX : log P( X\6) > 7 } (44) 

These level sets are used in deciding the class membership, even in the context of ML parameter 
estimation. We therefore estimate the parameters 8 to optimize an appropriate level set. As before, 
we cast this problem as MRE: 

Definition 3 Given a probability model P(X\9 ), 9 e 0, a set of training examples {Ah,..., At}, 
a set of margin variables 7 = [ 71 ,..., 7 t], and a prior distribution Po(8,y) we find the MRE 
distribution P(8,~/) such that minimizes D(P\\Po) subject to the constraints 

I P(8 , 7 ) [log P(X t \9) - 7i ] d9dj > 0 (45) 

for all t = 1 ,..., T. 

Note that this is again a MRE projection problem whose solution can be obtained as before. 
The choice of Po(j) in Po(8,j ) = Po(8)Po(j) is not as straightforward as before since each margin 
7 1 needs to be close to achievable log-probabilities. We can nevertheless easily find a reasonable 
choice e.g. by relating the prior mean of 7 1 to some a—percentile of the training set log-probabilities 
generated through ML or other standard parameter estimation criterion. Denote the resulting value 
by l a and define the prior P 0 ( 7 t) as Po( 7 t) = ce _c(/a_ " f) for 7 * < l a . In this case the prior mean 
of 7 1 is l a - l/c. 

6 The comparison is done upto a scaling factor and an additive constant. 


17 





Figure 9: a) Distribution of training set log-likelihoods for the MRE model (solid line) or the Bayes 
model (dashed-line), b) ROC curve for the two models on an independent test set. 

We have verified experimentally for a simple product distribution that this choice of prior to¬ 
gether with the MRE framework leads to a real improvement over standard (Bayesian) approach. 
Figure 9 illustrates the benefit of the MRE approach for discriminating between true and spurious 
splice sites. The examples were fixed length DNA sequences (length 25) and we used the following 
product distribution of simple multinomials: 

25 25 

P(X\6) = J] PiixilOi) = n 0*i\i (46) 

i -1 i =1 

where X = {aq,..., .T 05 }, aq £ {A, C, T, G}, and ^ x . 9 x m = 1- The model parameters {9 Xi u} were 
estimated on the basis of only true examples (7000). The estimation criterion was either Bayesian 
with an independent Dirichlet prior over each component distribution {9.^} or through the relative 
entropy projection method with the same prior. Figure 9a) indicates, as expected, that the training 
set log-likelihoods from the MRE method are more uniform and without the long tails 7 . This 
difference leads to improved anomaly detection as shown by the ROC curve in Figure 9b). The test 
set consisted of 1192 true splice sites and 3532 spurious ones. 

We expect the effect to be more striking in the context of more sophisticated models such as 
HMMs that may otherwise easily capture spurious regularities in the data. In the next section we 
describe how such models can be used efficiently within the MRE framework. 

4.1 Extension to latent variable models 

In the presence of latent variables (missing information) we can no longer use the above formula¬ 
tion directly. This arises because log P(X t \9) does not decompose into a sum of simple components. 
We can, however, achieve an efficient lower bound solution. If we let Xf, be the set of latent variables, 
we can resort to the following variational lower bound: 

log P(X t \9) >Y,Qt(X h )logP(X u X h \9)+H(Q t ) (47) 

x h 

where H(Q t ) is the entropy of the Q t distribution. A separate transformation has to be introduced 
for each training example. Note that the lower bound is reasonable in this context since the objective 

7 To compute these log-likelihoods from the MRE method, we used the MRE solution as the posterior distribution 
over the parameters. This is suboptimal for the MRE method given that the criterion is slightly different but suffices 
here for the purposes of illustration. An analogous figure with minor differences could be computed on the basis of 
f P(0) log P(X\0)d0 for the two methods. In this case, the figure would be suboptimal for the Bayesian approach. 
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is to guarantee that all (or most) training examples have likelihoods above some margin threshold. 
Whenever the lower bound exceeds the threshold, so does the original likelihood. 

The MRE distribution P(8, 7 ) is obtained under the following constraints: 


j P(0,l) 


J2Qt(x h )\ogP(x u x h \0) 

- X h 


-7 1 


dO + H(Q t ) > 0 


(48) 


which are of the same form (linear) as before. Note that we have made an additional assumption 
that Qt(Xy, ) is functionally independent of the parameters 6. This assumption guarantees that the 
MRE distribution P(0, 7 ) can be computed efficiently for a large class of probability models such 
as mixture models and HMMs. The loss in accuracy due to this simplifying assumption vanishes 
whenever the (marginal) MRE distribution P(8) becomes peaked. In principle, this means that we 
can always find the single most discriminative setting of the parameters even with the variational 
bound. Roughly speaking, we incur a loss only relative to the exact MRE approach. 

The overall solution to the MRE problem is no longer unique, however, but we can find a locally 
optimal solution iteratively as follows: 

Step 1. Fix {Qt( -Y/,)} and find the MRE distribution P(6, 7 ) as before 

Step 2. Fix P{6, 7 ) and let 


Qt(X h ) * exp | J P{6) log P(X U X h \0)d,0 J 


(49) 


Both steps can be computed efficiently for a large class of models such as HMMs assuming the prior 
Po(8 ) is Dirichlet and factorizes across the parameters. More generally, the prior should be the 
conjugate prior satisfying the parameter independence assumption of [ 6 ] (see also [9]). 

The iterative algorithm actually converges in the sense defined by the following theorem: 

Theorem 6 If we let P^{8 i ^) be the MRE distribution after n steps of the iterative algorithm 
described above , then 


D(P {1) |jP 0 ) > D(P {2) ||Po) > > D(P {n > HP,) (50) 

The theorem is easy to understand as follows: each time we optimize any of the Q t { dis¬ 
tributions, we maximize the associated lower bound. This maximization relaxes the corresponding 
constraint on the MRE distribution and allows the relative entropy to be decreased. 

5 Uncertain or incompletely labeled examples 

Examples with uncertain labels are hard to deal with in any standard discriminative classification 
method, probabilistic or not. Note the difference between labels that are inherently stochastic and 
those that are predictable but merely missing (the case considered here). Uncertain labels can be 
handled in a principled way within the maximum entropy formalism: let y = {y 1 ,... ,yr} be a set 
of binary variables corresponding to the labels for the training examples. We can define a prior 
uncertainty over the labels by specifying P 0 (y); for simplicity, we can take this to be a product 
distribution 


Po(y) = RpM 

t 


(51) 
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where a different level of uncertainty can be assigned to each example. We may, for example, 
set Ptfiiyt) = 1 whenever y t is observed and Pt,o(yt) = 0.5 if the label is missing. The TIRE 
solution is found by calculating the relative entropy projection from the overall prior distribution 
Po(0,%y) = Po(Q)Po('j)Po(y) to the admissible set of distributions V (no longer directly function 
of the labels) that are consistent with the constraints: 

V f P(0, 7 , y) [y t £(X t ,@) - 7 * } d@ dj > 0 (52) 

v Je H 

for all t = 1, ... ,T. The prior distribution Po(j) in this formulation encourages decision rules that 
achieve large classification margins for the examples (most of the probability mass is assigned to 
values 7 1 >0). This preference towards large margins creates dependencies between the (a priori) 
unknown labels and the parameters 0 of the discriminant function. Consequently, even unlabeled 
examples will contribute to the (marginal) MRE distribution P(0) that specifies the decision rule. 
We may alternatively view the MRE formulation as a transduction algorithm [22] whose objective 
is to determine the class labels for a set of unlabeled training examples. 

While this provides a principled framework for dealing with uncertain or partially labeled ex¬ 
amples, the MRE solution itself is not in general feasible to obtain. For example, in the context 
of support vector machines (for an alternative approach see [2]), the MRE distribution over the 
labels will be (roughly speaking) a Boltzmann machine and therefore not manageable in general via 
exact calculations. We can nevertheless employ efficient approximate methods to obtain an iterative 
algorithm for self-consistent probabilistic assignment of the uncertain labels. 


5.1 Feasible approximation 

To be able to deal with uncertain labels in a feasible way, we solve instead the following MRE 
problem with additional constraints: 

Definition 4 Given a parametric discriminant function £(X,@), a set of margin variables 7 = 
[ 71 ,... ,7 t], a set of class variables y = [y 1 ,... , and « prior distribution 


Po(®,7,y) 


Po(Q) 


RPoM 




(53) 


we find a constrained MRE distribution P(6,^,y) of the form P(6,^)P(y) that minimizes D(P\\Po) 
subject to the constraints 

V / P(0 ,7 )P(y) [ytAXt, 0) - 7 1 ] d 0 dj > 0 (54) 


for all t = 1,,.,, T, 


We may view this as a type of mean field approximate since the MRE distribution is forced 
to factorize to make the problem tractable. The solution is no longer unique but can be obtained 
through the following two-stage iterative algorithm: 


Step 1. Fix P(y) and let p t = P{y)y t - We find P(0,y) as the MRE solution subject to the 
constraints 


/ P(0, 7 ) [Pt£( A' t ,0) - 7 *] dQ d 7 > 0 (55) 

Je, 7 

Note that since the prior factorizes across {©, 7 } the MRE solution factorizes as well, i.e., 
P(0, 7 ) = P(0)P( 7 ). 
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Step 2. Fix the marginal P(0) obtained in the previous step and find the MRE solution P'(y, 7) 
subject to 

p '( y , 7 ) | J P(B) [y t C(X U 0) - 74 ] d0 J d 7 > 0 (56) 

for all t. Update P(y) f— (1 — e)P(y) + eP'(y) or simply set p t f— (1 — e)p t + ep' t where 
p't = T,yP'(y)yt- 

The fact that we include P(y) also in the second step is necessary since any adjustments to 
the labels must be compensated by an increased margin. The distribution P(y) is updated via 
relaxation to ensure a more controlled adjustment of the labels; any large change in P(y) is likely to 
induce a significant subsequent modification to the solution of the first step. Although the iterative 
algorithm remains stable even if larger changes are made, we believe the relaxation update leads 
to better local optima. Moreover, since the admissible set is convex and because the minimization 
objective (relative entropy) is also convex, the relaxation update always yields a change in the 
appropriate direction. The solution to either step is well defined and can be obtained in closed 
form assuming the problem is tractable when we have complete information about the labels. The 
iterative algorithm is well-behaved in the sense of the following theorem: 

Theorem 7 Let P (n) ( 0 , 7 , y) = p( n )(0,y)p( n )(y) be the constrained MRE solution after n itera¬ 
tions. Then for all 0 < e < 1, where e is the step size used in the algorithm, we have 

P(P (1) |jPo) > P>(P (2) ||P 0 ) > ... > P>(P (n) ||P 0 ) (57) 

The result holds also after either step of the two-stage iterative algorithm. 

5.2 Example: support vector machines 

Here we provide a preliminary numeral assessment of how the above algorithm is able to make use 
of unlabeled examples in the context of predicting DNA splice sites with support vector machines. 
A detailed formulation of the algorithm for SVMs can be found in Appendix C. We generated three 
training sets of examples corresponding to whether 1 ) all the labels were known, 2 ) labels were 
provided only for about 10% randomly chosen examples and the remaining 90% were unlabeled but 
available, and 3) only the 10% labeled examples were used for training. The full training set in this 
case consisted of 500 true DNA splice sites and 500 spurious ones (false examples). The examples 
were fixed length (25) strings of DNA letters (A,C,T,G) which were translated into bit vectors using 
a four bit encoding (e.g. A —> [1000]). Figure 10 gives ROC curves based on an independent test set 
(1192 true examples and 3532 false examples) for SVMs trained with one of the three training sets. 
Note that when the training set is fully labeled the algorithm reduces to the standard formulation. 
The figures show that even the approximate formulation 8 is able to reap most of the benefit from 
the unlabeled examples. The finding is also robust against the choice of the kernel function as is 
seen by comparing Figure 10a) and 10b). The findings are preliminary. 

6 Discussion 

We have presented a general approach to discriminative training of model parameters, structures, 
or parametric discriminant functions. The formalism is based on the minimum relative entropy prin¬ 
ciple reducing all calculations to relative entropy projections. Quite remarkably, we can efficiently 

8 In our experiments, e = 0.1 and the iterative algorithm was run for 10 iterations. The benefit may vary as a 
function of e and the number of iterations, particularly if e is too large. The prior probability Po(y) = Po,t(yt) 
over the labels were set to 0 or 1 when the label for yt was observed and to 0.5 for the unlabeled ones. 
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Figure 10: a) test set ROC curves based on a training set with fully labeled examples (solid line), 
90% unlabeled and 10% labeled (dot-dashed), only the 10% labeled examples (dashed). In a) a 
linear kernel was used and in b) a Gaussian kernel. 


and exactly compute the best discriminative distribution over tree structures within this framework. 
The MRE idea gives, in addition, a natural discriminative formulation of anomaly detection prob¬ 
lems or classification problems involving partially labeled examples. Efficient algorithms were also 
given to exploit such formulations. 
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A Computing averages under a factored distribution over tree structures 

Lemma 4 If P(E) is given by equation (34) and f,g are functions of E additive in the edges (i.e. 
f(E) = Euv&e fuv) then 


< f(E) > P 
< f(E)g(E) > P 


I 8\Q(We af )\ 

Z da <> u 

}_ c> 2 | Q(We af+f,g 

% dad/3 


a=(3=0 


(58) 

(59) 
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This lemma can be easily proved by equating \Q(We a f)\ with its definition (36) and then taking 
derivatives of both sides. Then, remembering that for any matrix .4 with elements Ay 


M. 

dAij 


I aka- 1 )^ 


(60) 


one obtains, after conveniently grouping the terms, the result of Lemma 5: 

Lemma 5 Let P(E ) and Q be given by equations (34) and (36) respectively, M be a symmetric 
matrix with 0 diagonal defined by 


f i[(Q '(uu + iQ 1 )w ~ 2(Q 1 ) ut ,)], u,v < n 
1 f(Q~ 1 )™. V < u = n 


(61) 


and f a function of the structure E satisfying f(E) = ^2 uveE fuv Then the average of f under P 
is 

n 

<f(E)> P = J2 P (E)f(E) = E fuvW uv M uv . (62) 

E u,v=l 


B Integrating over the parameters P(E S ,6 S ) 
Let us define 

N uv (Xw>%v) — 


K,” = 


E sXt y f X = E sXtVt 

t:v=x v ,u=x u t:v=x v 

F{N° v {x u ,x v ) + Nl v (x u ,x v )) 


nn 

X u 

n 


L(N° v (x u x v )) 
T(N((x v ) + N((x v )) 

T(N(M) 

With these notations we can express W£ v and Wq in equation (43) as 

T(N S ) 


W s = 

rr uv 


K S L? S 

u™v 


and Wn = 


T(N S + X s ) 


n- 

vev 


(63) 

(64) 

(65) 


( 66 ) 


In the above, T() denotes Euler’s Gamma function. Note that the “counts” N* v can be either 
positive or negative, so that the variables k may not be defined for arbitrary values of A. All the 
above expressions exist, however, for A = 0; in this case W£ v = Wq = 1. 

The classification rule is given by equation (39) with w s uv (X),Wq(X) redefined as 

<JK = nKv(^x v ) + K v (x u v- V )] - 9[NZ(*v) + KM] - nKM + KM] (67) 


<(K = E *IKM + KM] - m s + M ( 68 ) 

vev 

with ^ representing the derivative of the log-Gamma function: 

‘I'U) = fjogT(z) (69) 

Note the similarity with the fixed parameter case: the classification rule is still an average of a 
log-likelihood difference; the functions arise from averaging the log-likelihood under the MRE 
distribution of the 0 parameters. 
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C Uncertain labels and support vector machines 


We provide here more details about the two step feasible algorithm for dealing with partially la¬ 
beled examples in the context of support vector machines. We start by defining the prior distribution 
over all the parameters as 

P*{0,b,l,y) = Po(8)Po(b)Poh)Po(y) (70) 

where Po(8) is and Po(b) approaches a non-informative prior. By the non-informative prior 

we mean here a limit of Po(b\k) = A r (0,I ■ k ) as k —>■ oo. The prior over the labels is assumed to 
factorize across the examples, i.e., 

p 0 (y) = l[p 0 Ayt ) (7i) 

t 

where, for example, we can set each Po.tXyt ) = 1 whenever the corresponding label yt is known and 
P<o,t.(Vt) = 0.5, y t = ±1 for all unlabeled examples. We use here Poi'j) from eq. (9); the alternatives 
were discussed in the text. 

Let now p t = Yl y Po(y)yt = J^ yt Po,t(yt)yt, where p t is the mean value of the label. With these 
initializations, the two step algorithm is given as follows: 

Step 1. We fix {p t } and find the MRE solution for P{6,b, 7 ). Based on Lemma 3 P{6, 7 ) and P(b) 
can be found separately. For P(6, 7 ) the the Lagrange multipliers are obtained by maximizing 
(analogously to Theorem 2): 

Je, 7 (A) = £[ A t + 1 °g( 1 - V4 } ~ l E X t\t'PtPr [XjX r ) (72) 

t 1 t,t' 

subject to the constraint that Ylt X tPt = 0 - This is no more difficult to solve than the original 
SVM optimization problem with hard labels. 

As for the bias term b, we only need its mean relative to the MRE solution, i.e., b = f P(b)bdb. 
This can be computed as the limit of the means corresponding to proper priors Po(b\k) (each 
MRE solution P(b\k) based on P 0 (b\k) is a Gaussian with a well-defined mean). We omit the 
algebra and instead provide the answer in terms of the following averages: 

L t = f P(0 ) (8 T X t ) dd = Y, \vPt' {XjX v ) (73) 

t’ 

7 1 = f P{ 7) 7 1 d~f = 1 -“T - (74) 

J c- X t 

The desired mean b is now given by 

b = argmax | min( p t (L t + b) - 74 ) j (75) 

This setting optimizes the most critical constraints of eq. (55). In other words, b maximizes 
the minimum of the left hand sides of eq. (55). 

Step 2. To update the MRE distribution over the labels, we fix P(6,b) and find P'(y,”f) subject to 

V / p'(y, 7 ) / P(0, b ) [ yt(8 T Xt + b) - 7t ] dSdbdh 

y J J6,b 

= E / 7 ) [ VtiLt + b)~ Jt ] d 1 > 0 (76) 

y 
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Analogously to the first step, the Lagrange multipliers are found by maximizing the corre¬ 
sponding —logZ (algebra omitted): 

jy, 4 X) = E j x 't + 1 °g ( 1 - x 't / c ) - lo s E PM^ ytmt+ ~ b) ) (77) 

< l !H”. ' 1 J 

Note that the Lagrange multipliers here are not tied and can be optimized independently for 
each t. This happens because we have assumed that the prior distribution factorizes across the 
examples and because the discriminant function does not tie the variables together. Each of 
the one dimensional convex optimization problems are readily solved by any standard methods 
(e.g. Newton-Raphson). The resulting MRE distribution over the labels, P'(y ) is given by 

P'(y) = llPKvt) (78) 

t 


where 

Pl\yi) = ^PoAy i )e v ^ +l) 

At 

We can easily compute p' t = J P[{y t )yt from this result. Finally, the updates 

Pt (1 - e)pt + ep't 

complete the second step. 

The decision rule for a new example A' is given by 

V = sign ^ X tPti x I x ) + b 

where {A t } and b are the solutions to the first step of the iterative algorithm. 


(79) 


(80) 


(81) 
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